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METHOD AND SYSTEM FOR THREE-DIMENSK MAL HANDWRITING 

RECOGNITION 



5 

BACKGROUND OF THE INVEN ION 

The invention relates generally to handwriting r ignition technologies, and 

more particularly to three-dimensional (3D) handwritii i recognition methods and 
systems. 

10 

Handwriting recognition Is a technique by which an Intelligence system can 

recognize characters and other symbols written by han This technique has been 

very popular since it frees users from the keyboard, allow g users to write and draw in 

a more natural way. With the increasing demands from u ars, more and more devices 

15 now have incorporated handwriting recognition system x> give users natural input 

experience. Handwriting recognition is particularly p pular in various handheld 

devices that provides handwriting recognition of words v th complex structures, e.g., 

Chinese characters, input of which using a conventii lal keyboard Is very time 
consuming. 

20 

A handwriting input device provides a user with a friendly way of inputting 

information. At present* the minimum requirement for an lputtlng device is a mouse. 

To write with the mouse, the user needs to press and h« d down the primary mouse 

1 
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button, and then move the mouse pointer to form stroh s of a word or character for 

generating the final word or character Popular handwi ing input devices, such as a 

pen stylus and tablet, are used on conventional handhi i devices such as PDAs, or 

are connected to a computer through a serial or USB | >rt Handheld devices often 

5 use a pen stylus and a touch screen as a pad to alio* jsers to perform recognition 

functions. Most handheld devices, such as PDAs, are e .lipped with this kind of input 
device. 

Another kind of handwriting Input device include a pen that allows users to 

10 transfer data into a receiving unit, such as a cellular phc e, a PDA or a PC by simply 

writing or drawing in their own natural handwriting on We comfort and space of a 
regular piece of paper. 

At present, all conventional handwriting input dev es adopt a two-dimensional 

15 input method. Users have to write on a physical mediu i, such as a tablet, a touch 

screen, or a paper pad. This limits the choices available tor users. For example, if a 

user wants to write some comments during a present* on or a lecture, he or she 

would have to first find a physical medium, e.g M a table a paper pad, etc. This can 

cause much Inconvenience for the user while standi g fn the room, giving the 

20 presentation or lecture. Also, In a mobile environment, uch as in a car, a bus or a 

subway it would be very inconvenient to "write" on a phy* :al medium using a stylus. 



2 
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Therefore, there is a need to provide an impr< «d handwriting recognition 
system that gives users more flexibility and convenience nd frees the users from the 
physical medium required for two-dimensional handwritinj recognition. 
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SUMMARY OF THE INVE TION 

The present invention gives users more flex flity and enjoyable writing 

experience by allowing users to freely write words or c aracters in a 3D space In a 

touchless way, without requiring any physical medium su i as a pad or a tablet 



According to the Invention, there is provided a hi idwriting recognition system 

that comprises an input device and a recognition devk in communication with the 

input device. The input device Includes a three-dinner lonal (3D) motion detection 

sensor configured to generate 3D motion data in respc se to a 3D motion. In one 

10 embodiment, the motion detection sensor measures acc leration of the 3D motion in 

X, Y and 2 axial directions to generate the 3D motion ds i. The recognition device is 

configured to receive (e.g.. by wireless means) the 3C motion data from the input 

device and derive corresponding two-dimensional (5 ») Images for handwriting 
recognition, based on the 3D motion data. 

15 According to one embodiment of the invention, the 3cognition device calculates 

corresponding 3D coordinates based on the 3D motion d< a, constructs corresponding 

3D tracks based the 3D coordinates, and derives the co esponding 2D images from 

the 3D tracks by mapping the 3D tracks onto a 2D plane f handwriting recognition. 

Other objects and attainments together with a uller understanding of the 

20 invention will become apparent and appreciated by referring to the following 

description and claims taken In conjunction with the accon >anying drawings. 
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BRIEF DESCRIPTION OF THE OR iWINGS 

The Invention Is explained In further detail, a i by way of example, with 
reference to the accompanying drawings wherein: 

FIG. 1 shows a three-dimensional handwriting re ignition system according to 
5 one embodiment of the invention; 

Fie. 2 Is a flow chart diagram Illustrating a recogn on process according to one 
embodiment of the invention; 

FIG, 3A shows a 2D image of a Chinese chare ter derived by mapping 3D 
hacks onto a 2D projection plane; 

f> FIG. 3S shows a final result of handwriting recogn on process based on the 2D 

image in FIG, 3A; 

FIG. 4 shows an external design of a 3D handwrf \g Input device according to 
one embodiment of the invention; and 

FIG. 5 Illustrates how the input device may be mou ted. 

> Throughout the drawings, the same reference imerafs Indicate similar or 

corresponding features or functions. 



5 
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DETAILED DESCRIPTION OF THE PREFERR D EMBODIMENTS 

FIG. 1 shows a three-dimensional handwriting rec gnition system 10 according 

to one embodiment of the Invention. As illustrated, systei 1 0 comprises a handwriting 

input device 20, a recognition device 30 and an outpul levies 40. Input device 20 

5 includes a 3D motion detection sensor 22, a control cin lit 26 and a communication 

interface 28. Recognition device 30 includes a processo 32, a memory 34, a storage 

device 36, and a communication Interface 38. For s nplicity, other conventional 
elements are not shown in FIG. 1. 



10 In operation, a user moves input device 20 to freel; write words or characters in 

a 3D space, e.g., in the air. Motion detection sensor 2: detects the 3D motion and 

communicates the 3D motion data and a sampling rate :> recognition device 30 for 

handwriting recognition via a communication interface 26 such as Bluetooth, ZJgbee, 

IEEE 802.11, infrared, or a USB port. The sampling rs a may be a predetermined 

IS value set by an end user or a manufacturer based on fa. Drs such as the processing 

capability of the system. Alternatively, the sampling rate may be dynamically 

determined and adjusted based on, for example, the spee of the motion. This can be 

done, for example, by first determining the speed of the I itlal motion associated with 

the handwriting. Then the recognition device can dynamic lly adjust the sampling rate 

20 based on the speed of the motion at the last sampling poi I The higher the speed is, 

the higher the sampling rate will be, and vice versa, e dynamically adjusting the 

sampling rate, the accuracy of the handwriting recognition an be improved since only 
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the optimal number of sampling points will be used r constructing the word or 
character. Further, lower power consumption is needed. 

Based on the motion data and the sampling rate n :eived from input device 20, 
5 processor 32 calculates the corresponding 3D coordinate on the X, Y and Z axes and 
stores these coordinates in storage device 36. Using the calculated coordinates, 
processor 32 constructs the corresponding 3D tracks, "he 3D tracks will then be 
projected onto a 2D plane to form 2D images whlc will be recognized using 
conventional handwriting recognition software. The final -suit is displayed on output 
10 device 40. 

Since 3D writing is a continuous process, control rcuit 26 of input device 20 
provides a control signal to recognition device 30 vi interface 28 to mdicate 
separation of Individual words or characters upon receivi 3 a user provided external 
15 input. For example, the user may press a control button t cause control circuit 26 to 
generate the control signal after completion of writing a wo, or a character. 

According to a specific embodiment of the inverrtic . motion detection sensor 
22 detects the 3D motion by measuring the acceleration of 18 movement along the X, 
20 Y and Z axes. As an example, the piezoreslstfve-type t -axial accelerating sensor 
commercially available from Hitachi Metals, Ltd., Tokyo Japan, may be used as 
motion detection sensor 22. This accelerating sensor in th« form of an IC chip has the 
ability to simultaneously detect acceleration in the three a> 3 | directions (X, Y and Z\ 
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The senor is highly sensitive and shock resistant ar i is a very small and thin 

semiconductor typo 3 axial accelerating sensor. P sre information about this 

accelerating sensor is available on the following website http^/www.hitachi- 

metal8,cojp/e/prod/prod06/p06_10.html, which is hereby icorporated by reference. 

5 

FIG. 2 is a flow chart diagram illustrating a recog tion process 100 performed 

by recognition device 30, according to one embodiment the invention. In FIG. 2, 

recognition device 30 receives the 3D motion data (e.g. the acceleration data of the 

movement fn the X, Y and 2 directions) and the sampli g rate from input device 20 

10 (step 102). Based on the information received, j ocessor 32 calculates the 

corresponding 3D coordinates on the X, Y and Z axes f • each sampling point using 

the starting point of the movement as the origin (step 1 6). Each sampling point Is 

also used as a reference point for calculating the coordin; e$ of the following sampling 
point 

15 

Calculation of the 3D coordinates is continuous f performed based on the 

incoming 3D motion data until processor 32 detects rec ipt of a control signal (step 

112). The control signal indicates completion of writing i word or a character. The 

corresponding 3D tracks are constructed using the 3D cc rdlnates (step 116) and are 

20 then mapped onto a 2D plane (step 122). Thereafter, anventional 2D handwriting 
recognition is performed (step 126). 



8 
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In the above recognition process 100, to map the ID tracks onto a 2D plane at 

step 122, it is necessary to first find a proper 2D pro ctton plane. In a preferred 

embodiment of the invention, a proper 2D projection p me is separately derived for 
each word or character. 

5 

According to the geometry principles, a proper 2D .rejection plane is a plane, to 

which the sum of the distance square of each sampling aint is minimal. Assume the 

coordinates of » sampling points are known as follows: 

(*i»3'j>«j),(*a,j' a ,Zt)— (*„> >»,,,*„) , and the equa an of the plane Is 

10 Ax+By+Cz+^O, where W*+JrVC*0). Now it is necessary to determine the 

values of A,S>C,D. The distance from one point ( w , to the plane is given as 
. _ |^ 1 +fly 1 +Cg 1 +J> | » 

1 TF+B* +C ' 8Um rSpr ie " ted by F (*>*>C,B) is 

expressed as: 

KtA » r ™ - _ (A +3* +Df +(Ax, +B M +Cz + f +...+(4*+Jto,+a!.+Dr 



IS 



20 



The values of A,B,C,D can be determined using the Lagrange multiplication 

method as follows, which is described In Mathem tfcs Analysis by Ouyang 

Guangzhong. published by Fudan University Press in 19 ) in China, which Is hereby 
incorporated by reference. Thus, 

F(A.B,C.0)-F<{A,B,C,D)=(J X , +By, +Ct, +2J>»+e^ + By a Z»> +]fy p +Cs , +0f 
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under the constraint * + a » +C > =i . From this equal ,n. the following equation la 
derived: 

G(4,-».C,/>) = ^UJ>,C,Z>)+^ a + J? 8 + c 3 -l) , 
where A is s Lagrange multiplier, which is a constant. T * partial differential functions 
5 are now performed on G(A,B,C,D) with respect to A, B. , and D as follows: 

dG(A> B,C,D) 
a* 

86(4-8, C,Z>) 

as 

8G(A t B,C,D} 
8C 

From the above four equations, the following equat ns are obtained: 

+*^. +«o-o 
where equation (4) can be rewritten as: 



10 



(1) 

(2) 
(3) 
(4) 
(5> 



(6) 



15 



By Incorporating equation (6) into equations (1), (2). nd (3). the following is 
obtained: 



10 



PAGE W23 ' RCVD AT 9/23/2008 4:43:19 PM [Eastern Daylight Tone] ' SVfcUSPTMKRM/O 1 DN1S:2738300 ' CSfl):9H 3320515 s DURATION (mm-ss):02-34 



Sap-23-2008 17:06 



FronrPHILIPS ELECTRONICS ICS 



914-332-0615 



T-179 P. 014/029 F- 



WO 2004/029866 



7182003/004102 



(7) 



Thus, from the above equations, the values of A, B, Can D can be obtained. 



As an example, the following 3D coordinates for total of 13 sampling points 
5 are obtained for a Chinese character 

{(0,0,0),(0.49,-1 ,0.02),(1 ,0.0.03),(0,0.0.02),(2,-0.13.0.01 ), l.5,-0.14.0),(3.76 r 0.31 .- 

0.01),(2.74,-1.26,0.0l),(1.3e,-2.24.0),(2.5,-2.0.01),(1.746, ,O.02.).(2.-1.5.0.03),(1.876.- 
4.5,0.02),}. 

10 By using the Lagrange multiplication method describ. i above, the following are 
obtained: 



53.407 


-23.6725 


0.1911 " 




~A 




A' 


-23.672S 


36J2195 


-0.2084 


* 


a 


= -x 


B 


0.1911 


-0.2084 


0.0035 








C 



A 2 +B* +C*=l,and 



From the above equations, the values of A, B, C and D ar. determined as follows: 
A = 0.0045,5 - 0.0O23.C = 0.9999,2) = -0.01777. 



Thus, the equation of the 2D projection plane fs 
0.0045*+0.0023y-f0.9999r-.o.oi777 = 0. From the equa 9n of the projection plane. 

11 
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Ax+By+Cz+D m o, and the equation of a line that is ve Heal to the projection plane, 

' A = B ' - c ' , the following equations are derive- 

_ CO 3 + C a )*, - ^Cgy, + Cz. + 
A*+B*+C 3 

_ {A- + C 3 }y, ~B(Ax, + Cz, + Z» 

5 which can be used to obtain the corresponding 2D coon: mtes for each 3D sampling 

point. In this example, the following corresponding 2D CO. dinates are obtained: 

{(0.0001.0) t (0.49 r 1).(1,-2.0001),(0,0).(2 I -0.13).(3.5001 t -0. 4),(3.7601,-0.31).<2.7401,- 

1 .26),(1 .3801 ,2.24),(2.5001 .-2),(1 .746, 1 ).(2,-1 .5001 ).(1 .87 ,-4.5001 )}. 

Based on these 2D coordinates, the corresponding 2D im ge is projected onto the 2D 

10 projection plane. The result of this Chinese character is . own In RG. 3A. The final 

result Is generated by performing a conventional 2D nan. vriting recognition process, 
as shown In FIG. 3B. 



In the above, the Lagrange multiplication method is ised to obtain the values of 

15 A, B, C and D. Other methods such as linear regression n food may also be used. 

FIG. 4 shows an external design of a 3D hai Iwriting input device 200 

according to one embodiment of the Invention. As showr n FIG. 4, input device 200 

Includes a housing 210 that contains the electronics parts f the device {such as a 3D 

20 motion detection sensor IC chip), a control button 212 to allowing a user to input a 
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control signal to Indicate completion of writing a word or, .aracter, and a band 220 for 
mounting input device 200 on the user's finger. 

FIG. S illustrates how Input device 200 may be me nted. In FIG. 5, device 200 

is mounted on a finger 232 of a user's hand 230. By mou ing it on the finger, the user 

can simply move the finger to write any word or charac >r in a 3D space. The 3D 

motion data will be wireless* transmitted to a recogn ion device for handwrttlng 
recognition. 

According to the Invention, the input device and tr recognition device can be 

integrated together as a single unit (e.g.. a pen) that oper tes in the same manner as 

described above. The final recognition results wili be ansmltted to an Intelligent 
apparatus such as a PDA, a laptop computer, a PC, etc. 

The Invention can also be used on a 2D plane if the ser so chooses. In such a 

case, the coordinates calculated will be for 2D plane and ,e step of mapping the 3D 
tracks onto a 2D plane is omitted. 



While the invention has been described in onjunction with specific 

20 embodiments, it is evident that many alternatives, modific- ions and variations will be 

apparent to those skilled In the art in light of the foregoing ascription. Accordingly, ft 

is Intended to embrace ail such alternatives, modifications nd variations as fail within 
the spirit and scope of the appended claims. 
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WHAT IS CLAIMED IS: 

1. A handwriting recognition system, comprisin : 

an input device including a three-dimensional (3D) notion detection sensor that 

is configured to generate 3D motion data in response to a ;D motion; and 

a recognition device, in communication with the in Jt device, that Is configured 

to receive the 3D motion data and derive corresponding 1 ro-dtmensional (2D) images 
for handwriting recognition, based on the 3D motion data. 



for performing 2D handwriting recognition based on the 21 images. 

3. The system of claim 1 , wherein the recogniti ^ device Includes: 



2. 



The system of claim 1 f wherein the recogr ion device Includes means 



means for calculating corresponding 3D a irdlnates based on the 3D 



motion data; 



means for constructing corresponding : > tracks based the 3D 



coordinates; and 



means for deriving the corresponding 2D inn res from the 3D tracks. 
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4. The system of claim 3. wherein the derivm means includes means for 
mapping the 3D tracks onto a 2D plane for deriving th 2D Images for handwriting 
recognition. 

5. The system of claim 3, wherein the recogi Hon device includes means 
s for performing 2D handwriting recognition based on the 21 images. 

6. The system of claim 4, wherein the calci ating means calculates the 
corresponding 3D coordinates of each sampling point b sed on the 3D motion data 
and a selected sampling rate. 

7. The system of claim 6, wherein the recog tion device further includes 
10 means for dynamically adjusting the sampling rate based n the speed of the motion. 

8. The system of claim 6, wherein the derivln means includes means for 
deriving the 2D plane as a plane to which the sum of le distance square of each 
sampling point is minimal. 

9- The system of claim 3, wherein the input de ce further Includes a control 
15 circuit, responsive a user's command, that is configured o generate a control signal 
for transmitting to the recognition device to indicate com etion of writing a word or a 
character. 

10. The system of claim 3, wherein the motioi detection sensor measures 
acceleration of the 3D motion in X, Y and Z axial dlrectio $ to generate the 3D motion 
20 data. 

15 ' 
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11. The system of daim S, further comprising s output device for displaying 
final results of the handwriting recognition. 

12. The system of daim 1, wherein the input d< Ice further includes a control 
circuit, responsive a user's command, that is configure to generate a control signal 

5 for transmitting to the recognition device to indicate cor >letion of writing a word or a 
character. 

13. The system of daim 1, wherein the moth i detection sensor measures 
acceleration of the 3D motion in X, Y and Z axial direct! is to generate the 3D motion 
data. 

10 14 , The system of daim 1, wherein the input avice wirelessly transmits the 

3D motion data to the recognition device. 

15. The system of daim 1. wherein the recc nition device includes means 
for performing 2D handwriting recognition based on the D images. 

16. A computing system, comprising: 

15 a memory: 

an Input device including a three-dimensional (2 i) motion detection sensor that 
is configured to generate 3D motion data in response t a 3D motion: and 
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a recognition device, operably coupled to the m« nory and in communication 

with the input device, that i B configured to receive the ID motion data and derive 

corresponding two^Imensiona! (2D) images for handwrlti g recognition, based on the 
3D motion data. 

s 17. The system of claim 16, wherein the recogi Bon device includes means 

for performing 2D handwriting recognition based on the 2t images. 

18. The system of claim 1 6, wherein the recognr >n device includes: 
means for calculating corresponding 3D cc rdinates based on the 3D 

motion data; 

10 means ftw constructing corresponding 2 > tracks based the 3D 

coordinates; and 

means for deriving the corresponding 2D ima 3S from the 3D tracks. 

19. The system of claim 18, wherein the derivins Tieans includes means for 
mapping the 3D tracks onto a 2D plane for deriving the D images for handwriting 

15 recognition. 

20. A handwriting recognition method, comprising he steps of; 
generating 3D motion data in response to a 3D motfc • and 
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denvlng corresponding dimensional (2D) imaj s for handwriting recognition 
baaed on the 3D motion data. 

21. The method of ctaim 20, further comprisir the step of performing 2D 
handwriting recognition based on the 2D images. 

5 22. The method of claim 20, further comprising e steps of: 

calculating corresponding 3D coordinates based or the 3D motion data; 

constructing corresponding 3D tracks based the 3C joordinates; and 

deriving the corresponding 2D Images from the 3D acks. 

23. The method of claim 22, wherein the step c deriving includes mapping 
10 the 3D tracks onto a 2D plane for deriving the 2D images f • handwriting recognition. 

24. The method of claim 22, further comprising =, step of performing a 2D 
handwriting recognition based on the 2D Images. 

25. The method of claim 23, wherein the corre* onding 3D coordinates of 
each sampling point are calculated based on the 3D rr tJon data and a selected 

IS sampling rate. 

26. The method of claim 25, further comprisi j a step of dynamically 
adjusting the sampling rate based on the speed of the motk 
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27. Tne method of daim 25. wherein the step f deriving further Indies the 
step of deriving the 2D plane as a plane to which the , ,m of the distance square of 
each sampling point is minimal. 

28. The method of daim 22, further comprisi. , the step of generating an 
5 indication, based on a user's command, to indicate corr letion of writing a word or a 

character. 

29. The method of. claim 22, further compri ng the step of measuring 
acceleration of the 3D motion ,n X. Y and Z axial director and whe re ,n the 3D motion 
date are generated based on the acceleration of the 3D r *on in the X. Y and 2 axia. 

10 directions. 

30. The method of claim 24. further comorlsir a step of dispiaying final 
results of the handwriting recognition. 

31. The method of claim 20. further compris. g the steps of wiretessiy 
transmitting the 3D motion data and wirelessly receivi, , the 3D motion data for 

15 calculating the 3D coordinates. 

32. The method of ciaim 20. further comprisi 3 the step of measuring 
acceleration of the 3D motion in X. Y and 2 axia, directions ,nd wherein the 3D motion 
data are generated based on the acceleration of the 3D m, Ion In the X, Y and Z axial 
directions. 
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33. The method of claim 20, further compri ing the steps of wiretessly 
transmitting the 3D motion data and wirelessly receh ig the 3D motion data for 
calculating the 3D coordinates. 

34. The method of claim 20. further comprislr the step of performing 2D 
5 handwriting recognition based on the 20 images. 
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